Today, we’re going to build our own toolchain using crosstool-NG. There are so many cases that we want to build a particular toolchain for a specific embedded system development board. One MOST important reason for that is probably because the specific development board has limited resource to build upon, therefore, the time-consumption to build some software on the development board is VERY inefficient. In the other way around, cross compiling on the host PC is MORE efficient with respective to the time-consumption.
In this blog, for simplicity, we take Raspberry Pi 3B as our demo development board, for which we are building the cross compiler. Some references can be found at:
1. Installation
How to install crosstool-NG are thoroughly summarized at its official website. In my case, I had it installed under folder /opt/compilers/crosstool-ng. Let’s take a look:
1 | pwd |
And make sure /opt/compilers/crosstool-ng/bin is under environment variable PATH.
2. Configuration
Under any directory that you want to save your .config file, we can configure our target cross compiler.
1 | ct-ng menuconfig |
According to elinux: RPi Linaro GCC Compilation, we need to do the following selections:
- Paths and misc options:
- Try features marked as EXPERIMENTAL: ticked
- Prefix directory: input the full path where you want to save the built toolchains
- Number of parallel jobs: 4. Sorry that I’ve NO idea if this is the number of cores on Raspberry Pi 3B, but Raspberry Pi 3B does have 4 cores.
- Target options:
- Target Architecture: arm
- Default instruction set mode: arm
- Use EABI: ticked
- append hf to the tuple (EXPERIMENTAL): ticked
- Endianness: Little endian
- Bitness: 32-bit
- Emit assembly for CPU: cortex-a53
- Use specific FPU: vfp
- Floating point: hardware (FPU)
- Toolchain options:
- Tuple’s vendor string: rpi
- Operating System
- Target OS: linux
- Version of Linux: 4.20.8. Currently, I flashed Raspbian Buster with desktop and recommended software, 2019-07-10 onto my Raspberry Pi 3B, it comes with kernel of version 4.19.66, which is NOT in the list. That’s why I decided to select 4.20.8 and try my luck.
😝1
2➜ ~ uname -a
Linux raspberrypi 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux - Binary utilities:
- Binary format: ELF
- Version of binutils: 2.31.1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19➜ ~ apt show binutils
Package: binutils
Version: 2.31.1-16+rpi1
Priority: optional
Section: devel
Maintainer: Matthias Klose <doko@debian.org>
Installed-Size: 95.2 kB
Provides: binutils-gold, elf-binutils
Depends: binutils-common (= 2.31.1-16+rpi1), libbinutils (= 2.31.1-16+rpi1), binutils-arm-linux-gnueabihf (= 2.31.1-16+rpi1)
Suggests: binutils-doc (>= 2.31.1-16+rpi1)
Conflicts: binutils-multiarch (<< 2.27-8), modutils (<< 2.4.19-1)
Homepage: https://www.gnu.org/software/binutils/
Download-Size: 56.9 kB
APT-Manual-Installed: no
APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
Description: GNU assembler, linker and binary utilities
The programs in this package are used to assemble, link and manipulate
binary and object files. They may be used in conjunction with a compiler
and various libraries to build programs. - C-library:
- C library: glibc
- Version of glibc: 2.28
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25➜ ~ apt show libc-bin
Package: libc-bin
Version: 2.28-10+rpi1
Priority: required
Essential: yes
Section: libs
Source: glibc
Maintainer: GNU Libc Maintainers <debian-glibc@lists.debian.org>
Installed-Size: 3,015 kB
Depends: libc6 (>> 2.28), libc6 (<< 2.29)
Recommends: manpages
Homepage: https://www.gnu.org/software/libc/libc.html
Download-Size: 657 kB
APT-Manual-Installed: yes
APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
Description: GNU C Library: Binaries
This package contains utility programs related to the GNU C Library.
.
* catchsegv: catch segmentation faults in programs
* getconf: query system configuration variables
* getent: get entries from administrative databases
* iconv, iconvconfig: convert between character encodings
* ldd, ldconfig: print/configure shared library dependencies
* locale, localedef: show/generate locale definitions
* tzselect, zdump, zic: select/dump/compile time zones - C compiler:
- Show gcc versions from: GNU
- Version of gcc: 8.3.0
- gcc extra config: --with-float=hard
- Link libstdc++ statically into the gcc binary: tick
- C++: tick
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20➜ ~ apt show gcc
Package: gcc
Version: 4:8.3.0-1+rpi2
Priority: optional
Section: devel
Source: gcc-defaults (1.181+rpi2)
Maintainer: Debian GCC Maintainers <debian-gcc@lists.debian.org>
Installed-Size: 46.1 kB
Provides: c-compiler, gcc-arm-linux-gnueabihf (= 4:8.3.0-1+rpi2)
Depends: cpp (= 4:8.3.0-1+rpi2), gcc-8 (>= 8.3.0-1~)
Recommends: libc6-dev | libc-dev
Suggests: gcc-multilib, make, manpages-dev, autoconf, automake, libtool, flex, bison, gdb, gcc-doc
Conflicts: gcc-doc (<< 1:2.95.3)
Download-Size: 5,200 B
APT-Manual-Installed: no
APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
Description: GNU C compiler
This is the GNU C compiler, a fairly portable optimizing compiler for C.
.
This is a dependency package providing the default GNU C compiler.
After we save the configuration to file .config, we Exit the crosstool-NG configuration dialog.
3. Build
Please remember to:
unset LD_LIBRARY_PATHbefore building. Otherwise, you’ll meet some ERROR messages.mkdir ~/srcbefore building. Otherwise, whenever you tried to rerunct-ng build, you’ll have to download ALL required packages from scratch.
1 | ct-ng build |
This process may take a while. I’m going to sleep tonight. Continue tomorrow…
Alright, let’s continue today. Glad to know it’s built successfully.
😄
Let’s FIRST take a look at what’s under the current folder.
1 | ls |
And then, let’s take a look at what’s built under the specified destination folder.
1 | ls ~/....../CrossCompile/RPi |
Finally, let’s take a look at the version of our built cross compilers for Raspberry Pi 3B.
1 | arm-rpi-linux-gnueabihf-gcc --version |
Additional issue: It seems current crosstool-NG does NOT officially support Python. Please refer to my issue at crosstool-NG.
4. Compile/Build a Package with Generated Toolchain
For most of the packages nowadays, they are
- either supported by make:
./configure -> make -> make install - or supported by CMake:
mkdir build -> cd build -> ccmake ../ -> make -> make install
How to use our generated toolchain to compile/build our target packages?
- For the FIRST option, you can follow the crosstool-NG Using the toolchain.
- For the SECOND option, you are welcome to follow CMake Toolchains.
Today, we’re going to take the package flann as an example, which are to be built with CMake.
4.1 CMakeLists.txt Modification
4.1.1 CMake Toolchains - Cross Compiling for Linux
By following CMake Toolchains - Cross Compiling for Linux, we FIRST modify flann CMakeLists.txt a bit by adding the following lines before project(flann).
1 | cmake_minimum_required(VERSION 2.6) |
- CMAKE_SYSROOT specify the sysroot directory which emulate your target environment, here, Raspberry Pi 3B
- CMAKE_STAGING_PREFIX is where the STAGING results store, for the reason that final results may require to be built in multiple stages. You may refer to Linux From Scratch for further background knowledge about that.
- tools actually specify the building tool directory. Making sure all generated cross compiling tools are under folder
${tools}/bin.
4.1.2 Ignore hdf5
In addition, for the emulated Raspberry Pi 3B sysroot, hdf5 is NOT supported. Therefore, let’s simply comment out the following line in flann CMakeLists.txt.
1 | #find_hdf5() |
4.2 Cross Compile
Now, let’s start cross-compiling flann.
1 | mkdir build |
and press c and then t, you’ll see the cross compiling toolchains have been automatically configured as:
1 | BUILD_CUDA_LIB *OFF |
The LAST step before make is to modify some of the parameters accordingly, in my case:
- BUILD_PYTHON_BINDINGS: ON -> OFF
- CMAKE_CXX_FLAGS:
-I/usr/include(for lz4.h from my host Ubuntu 19.04) - CMAKE_C_FLAGS:
-I/usr/include(for lz4.h from my host Ubuntu 19.04) - CMAKE_VERBOSE_MAKEFILE: OFF -> ON
- CMAKE_INSTALL_PREFIX:
/usr/local/-> under which directory you want to install, which can be IGNORED for now
Now, press c, you’ll see:
1 | CMake Warning at CMakeLists.txt:99 (message): |
A warining of missing hdf5 is clearly reasonable and acceptable. Then press g.
Finally, it’s the time for us to cross build flann.
1 | make -j8 |
VERY Important:
- If you build flann from source on a Raspberry Pi 3B, your system is going to hang HERE, possibly due to lack of memory…
- Raspberry Pi 3B has 4 cores ONLY, but now you can use MORE cores on your host PC, which is clearly one advantage of using cross compiling.
Finally, you’ll see flann has been successfully cross compiled as:
1 | ....../flann/src/cpp/flann/algorithms/autotuned_index.h: In member function 'void flann::AutotunedIndex<Distance>::optimizeKMeans(std::vector<flann::AutotunedIndex<Distance>::CostData>&) [with Distance = flann::L2<double>]': |
You can now:
make installto install the built/generated libraries installed under CMAKE_INSTALL_PREFIX- copy and paste the built/generated libraries onto Raspberry Pi 3B and use it directly.
BTW: do NOT forget to install the header files.
5. Additional Issues
Multilib/multiarch seems to be problematic nowadays. Please pay attention to Multilib/multiarch. Some related issues are enumuated as the end of this blog.
- Multilib caveats from Official Notes on specific toolchain features
- fatal error: gnu/stubs-hard.h: No such file or directory from crosstool-NG
- fatal error: gnu/stubs-hard.h: No such file or directory from gcc-cross-compiler
- Build error: “features.h: No such file or directory” from risc-v tools
- crosstool-NG risc-v linux multilib issue
- raspberrypi toolchain issue
Alright, that’s all for today. Let me go to bed. Good bye…
😪
😴
